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ABSTRACT 



The Even Start Family Literacy Program is a national program 
designed to break the cycle of poverty and illiteracy by working with 
low- income families to improve adult literacy, parenting skills, and 
developmental and preschool readiness of young children in the family. The 
Monongalia County (West Virginia) Even Start Program is unusual in that home 
visits are the primary service delivery mechanism. A program evaluation 
during the 1997-98 program year determined the outcomes of the adult and 
parenting education components and evaluated the program's parenting 
education literacy materials. Thirteen adults were tested for adult education 
outcomes, and 32 were tested for parenting education outcomes. Results 
indicate that the program had some impact on parents' reading comprehension, 
moderate impact on parents' number operations, very large impact on parents' 
problem solving, and large impact on parents' total mathematics. The program 
had more impact in the mathematics area than in reading comprehension. The 
program had no impact on the adults' parenting efficacy, little impact on 
parental responsibility, some impact on parents' internal locus of control, 
and some impact on parents' self-esteem. The parenting education materials 
developed by program staff were found to be interesting, understandable, 
informative, and utilitarian, but poor in terms of attractiveness. 
Recommendations included implementing changes to increase reading 
comprehension, determining the reason for the low parenting efficacy 
outcomes, and improving the appearance of the parenting education materials. 
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INTRODUCTION 



The National Even Start Program 

The Even Start Family Literacy Program is a national literacy program that identifies and 
addresses the educational and developmental needs of targeted families. These developmental 
needs of eligible Even Start families are addressed in three different ways. First, the educational 
and literacy needs of the adults in the family are identified and addressed. Second, effective 
parenting skills for the adults are taught. Third, the developmental and preschool readiness needs 
of the children in the family are addressed with at-home and/or center-based early childhood 
education sessions. 

The national Even Start program began in 1989. Legislative authority for the program is in 
the Hawkins-Stafford Elementary and Secondary School Improvement Amendments of 1988 to 
the Elementary and Secondary Education Act of 1965. Specifically, the program is Part B, 
Chapter 1 of Title I in Public Law 100-297 (Tao, Swartz, St. Pierre, & Tarr, 1997). Two years 
later, the original Even Start legislation was amended with the passage of the National Literacy 
Act (Public Law 102-73). Reauthorization for the Even Start program was provided by the 
Improving America’s Schools Act as Part B of Title I. 



Emphasis on Families 

The purpose of the Even Start program is to “help break the cycle of poverty and illiteracy” 
by working with low-income families to improve their educational opportunities. The legislation 
spells out those educational opportunities as being adult basic education or adult literacy, 
parenting education, and early childhood education. Further, the Even Start legislation requires 
that these three educational areas be integrated “. . . into a unified family literacy program” (P.L. 
103-382, Sec. 1201, as cited in Tao et al., 1997, p. 2 ). The Even Start program, by legislative 
mandate, represents an innovative combination of adult literacy or adult basic education, 
parenting education, and early childhood education. 

Criteria for eligibility into the Even Start program was established in the legislation. To be 
eligible for participation in the program, families had to meet two criteria. Participants had to be 
(1) a parent or parents who (A) are eligible for participation in an adult basic education program 
under the Adult Education Act, or (B) are within the state’s compulsory school attending age 
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ranges; and (2) the child or children, from birth through age seven, of the parent or parents 
described in (1). The (B) part of the parent eligibility requirement were new in 1995-96, which 
expanded the Even Start program to include teen parents for the first time. 



Program Services 

Even Start program services provided to eligible families can be divided into two types. 
Those two types of services to parents and their children are (1) “core” educational services and 
(2) ancillary support services. As mandated in the Even Start legislation, the “core” educational 
services are in three interrelated areas of: 

(1) high-quality instructional programs that promote adult literacy, 

(2) high-quality instructional programs that empower parents to support the 
educational growth of their children, and 

(3) developmentally-appropriate early childhood educational services and 
preparation of children for success in regular school programs (P.L. 103- 
761, Sec. 1205). 

The legislation also requires that Even Start programs provide these core services to parents and 
children jointly and, also, to provide some home-based services. 

Even Start programs are required to provide support services to participating families. 
The twin support services of child care and transportation for the purposes of enabling parents 
and their children to participate in Even Start programs are named specifically in the law. Local 
programs typically provide a range of other support services to their participating families, 
including health care, vision and hearing screening, employment assistance, mental health 
referrals, cross-agency assistance, nutrition education, referrals for screening or treatment for 
chemical dependency, and public library card attainment assistance. 

States receive their Even Start funding based on a formula related to the allocation they 
receive from the federal government for their Title I Program. The states then solicit grant 
applications and make awards on a competitive basis. The law provides a portion of the state’s 
allocation for state-level administration and management. 
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The Monongalia County Even Start Program 



Program Site Information 

Monongalia County, West Virginia, is located in the north central part of the state, 
bordering the Peiuisylvania state line. The population of the county, according to the 1990 U. S. 
Census, was 75,599 in its 417.85 square miles, or 180.92 residents per square mile. The county 
seat is the city of Morgantown, the fifth largest city in West Virginia. Approximately 34% of the 
total county population lives in Morgantown, with the remaining 66% living in the outlying, 
mostly rural and mountainous communities (Even Start Staff, 1995, p. III-3). Morgantown also 
is the home of West Virginia University, the largest university in the state. 

The Monongalia County school system had 10,079 students in the 1996-1997 school year 
in grades K to 12. There are 28 schools in the system and 598 teachers (Quality Education Data, 
1996, p. 14). In 1995, 3,615 (35%) of the system’s children qualified for free and reduced 
lunches (Even Start Staff, 1995, p. III-3). Like most of the state, the vast majority (93%) of the 
school children are Caucasian. The predominant organizational arrangement for the 20 
elementary schools is K-5 or K-6, although there is one K-3 school, two K-4 schools, and one 
school with no kindergarten. There are two middle schools (one grades 5-8 and one grades 6-8), 
two middle or junior high schools of grades 7-9, one small grades 7-12 school of 415 students, 
two large high schools (one grades 9-12 and one grades 10-12) of over 1,100 students each, and 
one vocational-technical education center. The Monongalia County Even Start Program is 
housed in the Dorsey Education Center, a renovated school building that also houses the 
administration offices for the Title I Program, the Head-Start Program, the Early Head Start 
Program, and two Head Start classrooms. 

There is poverty, unemployment, and illiteracy in Monongalia County. The district’s 
average of low-income families is 34%. The total caseload of families receiving health and/or 
human services in the county was reported as 5,174 in July 1994 (Even Start Staff, 1995, p. III- 
3). The number of persons unemployed in 1995 was reported as 2,580, which placed the county 
in the top seven in the state in this category. Data from the 1990 U. S. Census show that 22,562 
adults in the county have not completed high school and, further, there are 12,003 persons 18 
years and over with less than a ninth grade level of education. The Head Start Program has 
identified 236 unserved families who meet Headstart’s income eligibility guidelines and could 
benefit from participating in the Even Start program. 
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Program Information 

The Monongalia County Even Start Program began in 1993 and has operated since then. 
The program consists of three interrelated approaches to address family literacy and to prepare 
children for entrance into schooling. These three approaches are adult education, parenting 
education, and early childhood education. 

The Even Start program in Monongalia County is designed to provide simultaneous 
education services to participating parents and their children. The program has been designed to 
inform and educate parents on how important their involvement is in their child’s education and 
then to teach parents how to achieve this goal. The Monongalia County program staff has 
received training in the Parents As Teachers (PAT) program and has adapted it for its parenting 
education curriculum. The PAT is a nationally-validated home-school partnership program 
designed for 0-5 year old children. Its purpose is to provide the training and curriculum to 
support parents in giving their children the best possible start in life (Even Start Staff, 1995, p. 
III-l). 



In addition to the PAT curriculum, the Monongalia County Even Start program has 
developed and implemented its own family literacy model. This locally-developed model 
emphasizes children’s literature in fostering cognitive and educational growth in the family unit. 
A unique component of this model is that the program has developed an extensive lending library 
of children’s books; developmentally-appropriate toys; and other learning materials such as 
crayons, paints, markers, and art paper. These literacy development materials are loaned to Even 
Start families during regular, planned home visits. For example, a young boy might choose a 
book such as Tommy The Truck to read and a Tonka truck might be brought for him to play with 
before the next home visit by the Family Educator (described below). 

The unique feature of the Monongalia County Even Start program differentiating it from 
most other programs is that it is primarily a home-based program. While the law requires that all 
Even Start programs conduct some home visits, in the Monongalia County Program, home visits 
are the primary service delivery mechanism. The four Family Educators work with their 
assigned families to develop and schedule a plan of service. Most families are visited every other 
week in their homes by the Family Educator. Typically, these bi-weekly home visits last about 
one and a half hours each, but vary according to the family’s needs and other factors. During 
these visits, the Family Educator provides meaningful instruction to both the child or children 
present and to the parent. Typically, a home visit begins with the collection of books and toys 
loaned during the prior visit, and then moves into a learning activity with the child or children 
that focuses on literacy. Next, while the children are occupied, the Family Educator discusses 
child development issues or addresses parenting concerns, such as discipline (Even Start Staff, 
1995, p. III-2). Even Start Staff reported making a total of 1,103 home visits in 1997-1998 and 
circulating 3,388 books (1998, P. III-31). 
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Figure 1 is a graphic depiction of the Monongalia County Even Start Program with its 
eight essential components. This graphic portrayal of the program is a product of the prior 
evaluation of the Monongalia County program (Meehan, 1997). 



Program Information at the End of 1997-98. 

The following paragraphs are drawn from the data supplied in staffs report (Even Start 
Staff, 1998) and serve to describe the program, its families, and their characteristics. 

In June of 1996 there were 43 families in the Monongalia County Even Start Program that 
participated in at least two of the program components (parenting education and early childhood 
education). The composition of families in the program are 27 (63%) couple households and 16 
(37%) single households. Collectively, these families have 82 children participating in the 
program. The average number of children for all families was 2.3, but there was a difference 
across family type in that the average number of children in the couple households was 2.5, while 
the average in the single households was 1.8 children. 

The distribution of the ages of the 82 children between 0 and 7 years of age was rather 
even at 8% to 15% for ages 1, 2, 4, 5, and 6. There were more children in the 4- and 5- year old 
category, respectively, between 0-1, followed by 1 1 and 10 in the 2- and 3- year category, 
respectively. There were 8 children in each of the 0-1-, 1-6-, and 7- year old categories. The 
number of 4-year-olds participating in the Even Start program decreased slightly from 19 in the 

1996- 1997 year to 14 in the 1997-1998 project year. The percentage of those 4-year-olds 
enrolled in Head Start is viewed as one measure of this program’s success. Staff figures show a 
steady increase in the percentage of 4-year-olds enrolled in Head Start from 30% in 1993-1994, 
to 57% in 1994-1995, to 67% in 1995-1996, to 83% in 1996-1997. There was a slight decline to 
71% in the 1997-1998 year. 

The average yearly income for all 43 families in the Monongalia County Program in 

1997- 1998 was $1 1,257. However, this varied greatly by family type. For couple households in 
the program, the average family income was $13,160; for single households, the average family 
income was $7,592. The family incomes ranged from a low of $2,412 up to a high of $28,000. 



Comprehensive Evaluation 

In the fall of 1996, program staff contracted with the Appalachia Educational Laboratory 
(AEL) in Charleston, West Virginia, to conduct an independent evaluation of the Even Start 




10 



staff Deve/i 



4 



6 




Figure 1. Graphic Depiction of the 
Monongalia County Even Start Program 
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program during its fotirth year of operation. This evaluation was more comprehensive than 
previous local evaluations and was designed to analyze much client outcomes raw data collected 
by staff, in addition to collecting new qualitative data about the program and its processes. The 
four objectives of AEL’s evaluation were: (1) to identify the essential components of the 
program and the implementation variation of each component, (2) to evaluate the implementation 
of each of the program’s components, (3) to assess the adult and parenting education outcomes of 
the program’s Sample Study adults, and (4) to assess the early childhood education of the 
program’s Sample Study children. 

The Monongalia County project was one of the 57 sites randomly selected to be in the 
special Sample Study conducted nationally by Abt Associates. More detailed participation and 
outcome data were collected from parents and children in the Sample Study and there also were 
additional criteria to be included in the sample. This dramatically reduced the Monongalia 
County subsample to 16 families out of the total of 62 in the program in 1996. Too, there was 
some attrition within the subsample. One parent (mother) and one child from each of the 1 6 
families were targeted for the expanded data collection. 

Data for the AEL evaluation were collected in a variety of ways, including program 
records, observation of events, face-to-face interviews, and seven participant outcome measures. 
For adults, the Comprehensive Adult Student Assessment System (CASAS) Life Skills test 
(1997) was used to assess reading and mathematics literacy, while the Home Screening 
Questionnaire (HSQ) assessed parenting skills (Coons, Gay, Fandal, Ker, & Frankenburg, 1981). 
For each child, the PreSchool Inventory (PSI) assessed school readiness skills (Abt Associates, 
1994) and the PreSchool Language Scale - 3 (PLS-3) measured language development 
(Zimmerman, Steiner, & Pond, 1992). The three PLS-3 scales were: Auditory Comprehension, 
Expressive Communications, and Total Language. These seven outcome measures were 
administered three times over the two treatment years: pretest, posttest (end of first year), and 
follow up (end of second year). Statistical analyses of the outcome measures were t-tests for 
dependent means by the matched pairs technique. Effect sizes were computed for each t-test 
followed by items or points per treatment month gained. Graphic displays of each client’s score 
on each measure were designed and published in the final report (Meehan, 1 997). 

Overall, AEL’s first evaluation (Meehan, 1997) concluded that the Monongalia County 
Even Start Program is a comprehensive, rural, home-based family literacy program that produced 
many positive impacts on the adults and children studied. It was concluded that the Monongalia 
County Program staff have designed and implemented an effective rural, home-based Even Start 
Program for its clients. 
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Objectives for this Evaluation 



The Monongalia County Even Start Program staff contracted with the AEL staff to 
conduct a second, independent evaluation of their program. This second evaluation was 
conducted during the 1997-98 program year and was focused on adult and parenting education 
products and outcomes. Two issues drove this second AEL evaluation: (1) the families not in 
the national Sample Study completed both a different adult education outcome measure and a 
different parenting education outcome measure, and (2) program staff developed and tested new 
parenting education literacy materials and needed them evaluated. 

The main purposes of this evaluation were (1) to evaluate the impact of the program on a 
sample of families in terms of the adult and parenting education outcomes and (2) to evaluate the 
program’s original parenting education literacy materials. Four objectives were designed to 
address these two purposes: 

1 . to assess the adult education outcomes of a sample of the adults, 

2. to assess the parenting education outcomes of a sample of the parents, 

3. to compare the adult and parenting education outcome measures used in this 

evaluation with those used in the prior evaluation (Meehan, 1997) and make technical 

and utility recommendations to program staff, and 

4. to evaluate the program’s original parenting education literacy materials. 
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METHODOLOGY 



Evaluation Design 



Generally speaking, this is a product and outcome evaluation. One portion of the 
evaluation was devoted to assessing original parenting education literacy materials developed by 
Monongalia County Even Start Program staff. A second portion of the evaluation was devoted 
to assessing the outcomes of the Even Start program on a sample of adults. The design of each 
portion is explained in the following paragraphs. 

In an effort to supplement parenting education literacy materials commercially available 
for use with adult clients, the Monongalia County Even Start Program staff developed four 
original learning guides that covered topics not addressed by other available materials . The four 
topics were designed to teach parents the importance of basic literacy skills for their children and 
also included related activities for parents to complete with their children. The topics of the 
materials were contained in their titles: Learning the Letters of the Alphabet, Learning What 
Print Means, Retelling Stories, and Predictable Books. Physically, each topic was covered in one 
8 1 / 2 " by 11" page, printed on both sides. The front side provided background information on the 
topic — usually in the form of several questions answered by one paragraph of copy. The back of 
the page contained suggested activities for the parent and child to complete, ranging in number 
from five to eight. 

The four parenting education literacy guides were developed by the Even Start program 
staff and a graduate student at nearby West Virginia University and completed in several stages 
of internal review and revision. The materials were then ready for field testing with a small 
sample of Even Start families in the 1997-98 program year, which was designed by the graduate 
student. As part of the field test, a learner verification protocol (described later in this section) 
was designed and administered for each literacy guide. The evaluation of these literacy guides 
was based on the completed learner verification protocols from the sample of Even Start family 
adults and an overall assessment by the evaluator. 

The second portion of this evaluation dealt with the assessment of the outcomes of the 
Even Start project on a sample of adults. This is the second evaluation to assess adult outcomes 
of the project. The first effort (Meehan, 1997) was a comprehensive process and outcomes 
evaluation of a subsample of the program’s families who were in the national Sample Study 
conducted by Abt Associates (Tao, Swartz, St. Pierre & Tarr, 1997). This first evaluation 
included a detailed analysis of pretest, posttest, and follow-up test data from two instruments for 
each of the target subjects of parent and child in each of the 16 families in the Sample Study. 
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This evaluation concerns the pretest-posttest assessment of the outcomes of the Even 
Start program on adults not in the Sample Study. Families not eligible for participation in the 
Sample Study (based mainly on their enrollment date) completed a different set of adult 
assessment instruments. Even Start staff were interested in an analysis of the outcomes measures 
from those two different instruments plus some judgement of their technical and utility 
properties. The parenting outcome measure used with the national Sample Study adults had only 
a single total score, while the parenting outcome measure used with the other adults in the project 
had four separate subscale scores. The Even Start staff were very interested in an assessment of 
technical qualities of this new parenting education outcome instrument along with the actual 
parenting outcomes. 

It should be noted that the main difference between the two subsamples — ^those in the 
national Sample Study and those not in it — rested in the different outcome measures they 
completed — not the treatment they received. The home-based family literacy education services 
were the same for all families in the Monongalia County Even Start Program. 



Sample Families 



The Monongalia County Even Start Program was one of the 57 randomly-selected 
projects in the nation to be in the special Sample Study conducted by Abt Associates for the 
period of the 1993-1997 program years (Tao et al., 1997). Participating families in the Sample 
Study had to meet special criteria to be included in that effort and they completed additional 
outcome assessment instruments. The chief criterion for inclusion in the Sample Study was that 
the family first enrolled in Even Start and family members completed their pretests between 
September and November of the program year. However, many families enrolled in the program 
outside of that window of time, and, thus, were ineligible for the Sample Study. Of the 62 
families enrolled in the Monongalia County Even Start Program, only 16 were in the Sample 
Study (Meehan, 1997). The remaining families completed different outcome assessment 
measures from those in the Sample Study. These families were the subjects of this evaluation. 

Not all of the families completed the outcome measures for this evaluation and not all 
family members completed both the pretests and posttests of each instrument. Also, there was 
some attrition of enrolled families out of the local program for a variety of reasons. Within each 
family, one adult and one child were targeted to complete the outcome instruments. In this 
evaluation, all of the adults who completed the outcome instruments were female. Also, the 
number of Even Start adults with both pretest and posttest scores varied across the two 
instruments. In the end, there were 13 adults with both pretest and posttest scores on all four 
scales of the ABLE instrument and 32 adults with both pretest and posttest scores on the Parent 
Index instrument. 
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Data Collection Techniques 



Data for this evaluation of the Monongalia County Even Start Program were collected 
several ways, including program records, paper-and-pencil learner verification and outcome 
measures, and evaluator judgement. 



Program Records 

The program records used in this evaluation were the project-developed ABLE Scores 
and Parenting (sic*) Index summary sheets. This was a one page form that contained two major 
sections — one for each adult instrument. For the ABLE scores section on top of the page, the 
column headings were: scale name, date of administration, level, number of items, raw score, 
scale score, and grade equivalent. There were two rows for each scale, corresponding to pretest 
and posttest administrations. 

For the Parenting Index section on the bottom of the page, the column headings were date 
and the names of the four Parent Index scales. There were two rows for each scale, 
corresponding to pretest and posttest administrations. These cells were completed with the 
resultant raw scores by the Family Educators. However, it should be noted that these scores were 
not used in this evaluation because one of the objectives of the study was to assess the Alpha 
reliabilities of this new parenting education outcome measure. Given this need, the evaluator 
requested and received copies of all the Parent Index pretests and posttests completed by the 
parents in the project. Databases were constructed for these instruments, the raw data were 
entered, and those files were analyzed to yield descriptive statistics, total scores, and Alpha 
reliabilities. 



ABLE Instrument 

The Adult Basic Learning Examination (ABLE) is a set of tests designed to measure the 
educational achievement level of adults (Karlsen & Gardner, 1986a, 1986b, 1986c). The ABLE 
tests can be used to assess the achievement level of adults with varying amounts of schooling 
experience, but the test items are of adult content. These tests were developed as an instrument 
for adults who have not completed 12 years of formal schooling, but may be in an educational 
program designed to raise their educational levels. The ABLE tests cover the basic areas of 
reading, mathematics, and language arts. 



*as in the original developed by the program staff, changed from Parent Index 
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There are 6 subtests in the ABLE battery: vocabulary, reading, comprehension, spelling, 
language, number operations, and problem solving. Also, the two mathematics subtests are 
added to yield a total mathematics score. The number of individual items per subtest varies by 
the ABLE levels. For Level 1 , the number of items varies from 20 (both mathematics subtests) 
to 40 (reading comprehension) while for both Levels 2 and 3 the number of items varies from 30 
(spelling and language) to 48 (reading comprehension). Monongalia County Even Start staff 
decided to use four ABLE scales from the available set: reading comprehension, number 
operations, problem solving, and total mathematics. All three ABLE test levels were used in this 
evaluation, although Level 1 was used much less than Levels 2 and 3. Physically, the ABLE tests 
are in 8 3/8" x 10 7/8" booklets, printed in two colors (black plus green, blue, and red for each 
level, in order). These test booklets are reusable by the program. 

Reliability and validity information for the ABLE tests is provided in the norms booklets 
for each level (Karlsen &. Gardner, 1986a, 1986b, 1986c). Summarizing across the three levels, 
the Kuder-Richardson 21 internal consistency reliabilities for the four selected scales were: 
reading comprehension, .87 to .91; number operations, .82 to .88; problem solving, .81 to .94; 
and total mathematics, .89 to .93. No stability reliability (test-retest) data were provided in the 
norm booklets. In terms of validity of the ABLE tests, both intercorrelations among the ABLE 
subtests and correlations of the subtests with those of the Stanford Achievement Test were 
presented in the norms booklets. Again, summarizing across the three levels, the 
intercorrelations among the subtests were low to moderate, ranging from .45 (reading 
comprehension with problem solving in Level 1) to .71 (number operations with problem solving 
in both Levels 2 and 3). The correlations between corresponding subtests of ABLE and the 
Stanford Achievement Test across the three levels were moderate to high, ranging from .51 
(number operations) to .81 (total mathematics). 

As is usual with a well known, standardized achievement test series, several types of 
scores can be obtained and used from the ABLE series. The Monongalia County Even Start 
Program staff chose to record the objective raw score, the scaled score, and the grade equivalent 
score for each ABLE subtest administered. Once the objective raw score is converted to a scaled 
score, the administrator or analyst no longer needs to be concerned with which level or form of 
ABLE was completed. The scaled scores differed by the number of items in the subtest, ranging 
from 430 (reading comprehension. Level 1) to 885 (total mathematics. Level 3). 



Parent Index 

The Parent Index (PI) is an instrument developed in 1995 by researchers at the National 
Center for Family Literacy in Louisville, Kentucky. They developed the PI to use in evaluating 
parenting outcomes of their family literacy programs in several cities in the United States (J. 
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Tucker, personal communications, August 14, 1998). The Parent Index is a combination of three 
previously-developed scales designed to measure three constructs: parenting locus of control, 
internal locus of control, and self-esteem. 

The Parenting Locus of Control Scale was developed by Campis, Lyman, and Prentice- 
Dunn (1986) and consists of two separate scales: Parenting Efficacy and Parental Responsibility. 
Each scale consists of ten items and has the same Likert-type response scale of 1 (Strongly 
Disagree) to 5 (Strongly Agree). Physically, after space for general identifying information 
(program city, site, name, date, and social security number) and instructions appear on the top 
half of the first page of the PI, the ten Parenting Efficacy statements start and then continue on 
the second page, followed by the ten Parental Responsibility items. 

The Internal Locus of Control Scale was developed by Levenson (1981) to measure the 
perception of internal control the respondent feels over several situations compared to the 
amount of control external factors or forces have on those situations. This scale consists of eight 
items (none reversed). The response options for these items consist of a 6-point Likert scale: (1 
= Strongly Disagree, 2 = Disagree, 3 = Slightly Disagree, 4 = Slightly Agree, 5 = Agree, and 6 = 
Strongly Agree). The eight internal control statements fill out the top half of the third page of the 
PI. 



The Self-Esteem Scale was developed by Rosenberg in 1965. This scale consists of ten 
items and utilizes the same 6-point Likert scale as the Internal Locus of Control scale. In contrast 
though, five of the self-esteem items are reversed. These items complete the bottom half of the 
PTs third page. 

There is little reliability and validity information on the Parent Index instrument in its 
present, combined scales form. Some information was available in an unpublished Parent Index 
information sheet sent to the Monongalia County Even Start staff in response their request to 
review the instrument (J. Tucker, personal communications fax to Angie Swisher, June 6, 1997). 
The PI information sheet states that the Rosenberg Self-Esteem Scale was pilot tested in May of 
1995 and yielded reliability coefficients ranging from .64 to .81 . This lack of reliability data 
about the other scales of the PI instrument led to the discovery of such information being one of 
the objectives in this evaluation. 

Only the objective raw scores were used in this evaluation, as no norming data or 
transformed scores information was available for the Parent Index. 
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Learner Verification Protocols 

Four learner verification protocols were developed, one each for the topics of Learning 
the Letters of the Alphabet, Learning What Print Means, Retelling Stories, and Predictable 
Books. These protocols were completed by a small sample of adults in the Even Start program, 
although this sample for the field test of these new materials was not related in a systematic 
manner to the sample who completed either of the two instruments named above. 

Each learner verification protocol was printed on one side of 8 '/ 2 " x 11" paper and 
contained seven to nine items. The items on each protocol consisted of constructed response, 
open-ended items; one selected-response, closed-ended item; and one combination of the two 
types. Thus, across the four learner verification protocols, the vast majority of the items were the 
constructed response, open-ended type. 

The items in the learner verification protocols were designed to collect data related to the 
four areas of interesting to adults, understandable to them, informative to them, and utility to 
them and their children. Based on experiences with evaluating parenting learning materials 
developed at AEL and used in 47 states, the evaluator added attractiveness as a final area to 
assess the parenting education literacy materials. This area included evaluator judgements 
regarding the color of paper, the typestyle used, the layout of the materials, and graphics / 
illustrations. 



Data Analysis Procedures 



Treatment Months 

Analysis of the treatment months was rather straightforward. The ABLE Scores and 
Parenting Index summary sheets contained the pretest and posttest administration dates for each 
subtest/scale of those two instruments. In the SPSS-PC+ database, administration date variables 
were made for each subtest/scale and these dates were entered into the database along with 
instruments’ scores (described below). During data analysis, a new variable for the difference of 
the posttest date minus the pretest date, in months, was created for the ABLE and Parent Index 
instruments. Instrument administration dates on the locally-provided summary sheets were 
rounded to the whole month with the 15th day as the midpoint. These computations were done 
by hand and double checked for accuracy. These two, new treatment months variables were 
analyzed for descriptive statistics. 

Since the evaluation of the parenting education literacy materials was a point-in-time 
field test, treatment months were not relevant. 
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Treatment Effects 

The first step in analyzing the treatment effects for the parents was to transfer the ABLE 
and Parent Index scores from the summary sheets to the SPSS-PC+ database. There was 
provision for both pretest and posttest scores for the four ABLE scales and the four Parent Index 
scales for each adult. 

The second step was to create grade equivalent difference score variables for the four 
ABLE scales, compute those difference scores, and save them as the new variables. The third 
step consisted of computing descriptive statistics for the eight parent outcome scores, the two 
treatment periods (ABLE and PI), and the grade equivalent difference scores. The fourth step 
was to complete Cronbach Alpha reliabilities for the four PI scales, and design a table. 

For the fifth step, t-tests for dependent means by the matched pairs technique were 
computed for the four ABLE scores and the four PI scores. In the sixth step, data for computing 
the effect sizes for each t-test were input into the effect size formula developed by Cohen (1997). 
Here, the posttest mean score was subtracted from the pretest score and the remainder was 
divided by the standard deviation of the pretest score. Cohen’s effect size descriptors for t-tests 
were used to describe the numerical results. Seventh, and last, scale-specific tables were 
designed to display the PI Alpha reliabilities and the t-test and effect size data. 



Parenting Education Literacy Materials Evaluation 

An AEL evaluator analyzed the learner verification protocols completed by the parents. 
The open-ended responses were read and analyzed for their content. Here, the purpose was to 
develop themes and/or categories from the content provided. Then, the number of occurrences 
for each category was tallied. 

The closed-ended and the combination items were very similar in that they asked the 
respondents to check the numbered activities that they thought they would complete with their 
child. The combination item differed in that it asked which activity they thought they would not 
complete with their child. A probe then asked the respondent, “Why wouldn’t you do this?” 
Tallies of the number of checks per each numbered activity were made, followed by an analysis 
of the responses to the probe (if they were provided). 

The evaluator wrote a report first evaluating each of the four parent literacy materials, 
followed by a summary of them related to the four criteria of interesting, understandable, 
informative, and utility. In this summary, the evaluator judged the materials on their 
attractiveness, as well. 
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EVALUATION FINDINGS 



Treatment Months 



The number of treatment months differed greatly for the two parent outcome instruments. 
The ABLE treatment period spanned several program years while the Parent Index pretests and 
posttests were administered only during 1997-98 program year. Treatment month statistics were 
computed only for those adults who completed both the pretest and posttest for an instrument. 

The number of treatment months for the 1 3 adults completing both administrations of the 
ABLE instrument ranged from 5 to 42. The mean number of treatment months was 16.46, with a 
large standard deviation of 8.70. Thus, the average treatment period for the ABLE instrument 
was about two program years (program year is from September through May), but there was 
much variation across the sample of adults. 

The number of treatment months for the 32 adults who completed both administrations of 
the PI instrument ranged from 2 to 9. The mean number of treatment months was 6. 1 6, with a 
standard deviation of 1 .74. Thus, the average treatment period for the PI instrument was a little 
over six months. 



ABLE Instrument Outcomes 



Table 1 displays the results from the pretest-posttest administrations of the ABLE 
instrument with its four scales. Thirteen Even Start parents completed both pretest and posttest 
scores for the t-test analysis. 

The Reading Comprehension scale results are presented first in Table 1. The possible 
scale score points could range from 430 to 813. The pretest mean was 685.00 with a standard 
deviation of 61 .99. The coefficient of variation (the standard deviation divided by the mean) 
was .090 and the standard error of the mean was 17. 19, the largest in the table. The posttest 
mean was 702.62, a 17.62 gain from the pretest, and its standard deviation decreased to 46.89. 
Similarly, the posttest coefficient of variation decreased to .068, while the standard error of the 
mean also decreased more than four points to 1 3.00. At 2.08, the t- value was significant at the 
.10 level. The effect size for the Reading Comprehension scale was .284 or “small,” according to 
Cohen’s (1977) scheme. 
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The Number Operations scale results are presented next in Table 1 . The possible score 
on this scale could range from 477 to 885. The pretest mean was 659.85, with a standard 
deviation of 33.50. The coefficient of variation was .051 and the standard error of the mean was 
9.29, the smallest in the table. The posttest gained 2 1.61 to 681.46, while the standard deviation 
increased slightly to 34.24. The coefficient of variation was very similar at .050 to the pretest 
value, while the standard error of the mean increased slightly to 9.50. The t-value of 2.74 was 
significant at the .05 level. The Number Operations scale effect size was .645 or “medium.” 

The Problem Solving scale results are displayed third in Table 1. The scores on this 
scale could range from 477 to 885. The pretest mean was 674.62, with a standard deviation of 
47.68. The coefficient of variation was .071 and the standard error of the mean was 13.22. The 
posttest mean increased to 707.15 (at 32.53, the largest gain of the four ABLE scores) while the 
standard deviation decreased to 43.46. The coefficient of variation decreased slightly to .061 
and the standard of error of the mean decreased more than a point to 12.05. The t-value was very 
large at 5.02 and was significant at the .001 level. The Problem Solving scale effect size was 
.682 or “medium.” 

The Total Mathematics scale results are presented last in Table 1. This scale’s scores 
could range from 477 to 885. The pretest mean was 669.3 1, with a standard deviation of 35.27. 
The coefficient of variation was .053 and the standard error of the mean was 9.78. The posttest 
mean was 693.38, a 24.07 gain over the pretest, while the posttest standard deviation increased 
very slightly to 35.56. The coefficient of variation decreased a trivial amount to .051 while the 
standard error of the mean increased slightly to 9.86. The t-value of 3.98 was significant at the 
.01 level. And, similar to the two previous scales’ effect size, it was .682, or “medium.” 



Parent Index Outcomes 



Table 2 displays the Cronbach Alpha reliabilities for the four scales in the Parent Index 
instrument at both pretest and posttest. A total of 32 Even Start parents completed all the PI 
items in both administrations during the 1997-98 program year. For the ten-item Parenting 
Efficacy scale, the pretest Alpha was .68, while at posttest it was .70. The pretest Alpha for the 
ten-item Parental Responsibility scale was .73 and its posttest Alpha was .83. For the eight-item 
Internal Locus of Control scale, the pretest Alpha was a low .58, while the posttest Alpha 
increased to .71. Finally, the pretest Alpha for the ten-item Self-Esteem scale was .81 and it 
decreased to .75 at posttest. Thus, the Alphas increased on three of the four PI scales from pretest 
to posttest. 
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Table 1 

ABLE Instrument Subscale t-Test and Effect Size 
Results Over the Treatment Period 



Test 

Administration 


Number 

of 

Pairs 


Mean 


Standard 

Deviation 


Coefficient 

of 

Variation 


Standard 

Error 

of 

Mean 


t- Value 


Probability 


Mean 

Score 

Difference 


Effect 

Size 


Reading Comprehension 


Pretest 


13 


685.00“ 


61.99 


.090 


17.19 


2.08 


.059* 


+ 17.62 


.284* 


Posttest 




702.62 


46.89 


.068 


13.00 










Number Operations 


Pretest 


13 


659.85'’ 


33.50 


.051 


9.29 


2.74 


.018*' 


+21.61 


.645** 


Posttest 




681.46 


34.24 


.050 


9.50 










Problem Solving 


Pretest 


13 


674.62'’ 


47.68 


.071 


13.22 


5.02 


.00 r*” 


+32.53 


.682** 


Posttest 




707.15 


43.46 


.061 


12.05 










Total Mathematics 


Pretest 


13 


669.31'’ 


35.27 


.053 


9.78 


3.98 


.002‘” 


+24.07 


.682** 


Posttest 




693.38 


35.56 


.051 


9.86 











“ Reported in scale score points, which range from 430 to 8 13. 
^ Reported in scale score points, which range from 477 to 885. 

* Significant at .10. 

** Significant at .05. 

♦♦♦ Significant at .01. 

♦ Significant at .001 . 



# Effect size = small. 

# # Effect size = medium. 
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Table 2 

Alpha Reliabilities for the Four Scales in the Parent Index 
Instrument at Pretest and Posttest (N = 32) 



Sub scale Name 


Number 

of 

Items 


Alpha Reliability 


Pretest 

Administration 


Posttest 

Administration 


Parenting Efficacy 


10 


.68 


.70 


Parental Responsibility 


10 


.73 


.83 


Internal Locus of Control 


8 


.58 


.71 


Self-Esteem 


10 


.81 


.75 



Table 3 displays the results from the pretest-posttest administrations of the Parent Index 
instrument with its four scales. Thirty-two Even Start parents had scores on both administrations 
and were used in the matched pairs t-tests. 

The Parenting Efficacy scale results are displayed first in Table 3. The possible points 
on this scale were from 10 to 50. The pretest mean score was 40.25 with a standard deviation of 
4.39. The posttest mean score dropped a little to 39.69, while the standard deviation increased to 
5.94. This .56 drop in the mean was the only decline in the pretest to posttest scores for the eight 
scales of the two parent outcome instruments. The coefficient of variation at pretest was .178 
and the standard error of the mean was .78. At posttest, the coefficient of variation decreased 
slightly to . 1 50 (the smallest such value in the table), while the standard error of the mean 
increased to 1.05. The t-value was very small at -.59, which was not significant at the .10 level. 
The Parenting Efficacy scale effect size was -.128. Cohen’s (1997) descriptors for t-test effect 
sizes start at the .20 value and is labeled “small.” Thus, we label this effect size of -.128 as “less 
than small.” 
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Table 3 

Parent Index Instrument Subscale t-Test and Effect Size 
Results Over the Treatment Period 



Test 

Administration 


Number 

of 

Pairs 


Mean 


Standard 

Deviation 


Coefficient 

of 

Variation 


Standard 

Error 

of 

Mean 


t- 

Value 


Probability 


Mean 

Score 

Difference 


Effect 

Size 


Parenting Efficacy 


Pretest 


32 


40.25“ 


4.39 


.178 


0.78 


-0.59 


.561 


-0.56 


to 

00 


Posttest 




39.69 


5.94 


.150 


1.05 










Parental Responsibility 


Pretest 


32 


30.03“ 


6.29 


.209 


1.11 


0.82 


.419 


+0.81 


.129" 


Posttest 




30.84 


7.31 


.237 


1.29 










Internal Locus of Control 


Pretest 


32 


32.32*’ 


5.37 


.165 


0.95 


1.99 


.056* 


+1.62 


.304"" 


Posttest 




33.94 


5.68 


.167 


1.00 










Self-Esteem 


Pretest 


32 


42.31' 


9.86 


.233 


1.74 


2.22 


.034" 


+1.85 


00 

00 


Posttest 




44.16 


9.50 


.215 


1.68 











® Possible points = 10-50. 
^ Possible points = 8-48. 
® Possible points = 10-60. 



* Significant at .10. # Effect size = less than small. 

* * Significant at .05. # # Effect size = small. 
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The Parental Responsibility scale results are presented second in Table 3. As above, the 
possible scale scores were from 10 to 50. The pretest mean was 30.03, with a standard deviation 
of 6.29. The pretest coefficient of variation was .209 and the standard error of the mean was 
1.11. The posttest mean increased slightly to 30.84 (a .81 gain) and the standard deviation also 
increased to 7.3 1 . The posttest coefficient of variation was .237 (the largest such value in the 
table), while the standard error of the mean increased slightly to 1 .29. The t-value was very 
small at .82 and it was not significant at the .10 level. At .129, the Parental Responsibility effect 
size was under .20 and, thus, was labeled “less than small.” 

The Internal Locus of Control scale results are the third ones to be displayed in Table 3. 
The range of possible scores was from 8 to 48. The pretest mean was 32.32, with a standard 
deviation of 5.37. The pretest coefficient of variation was .165 and the standard error of the 
mean was .95. At posttest, the mean increased 1.62 points to 33.94, while the standard deviation 
increased slightly to 5.68. The coefficient of variation increased a trivial amount to .167, while 
the standard error of the mean increased to 1 .00. The t-value was 1 .99, which was significant at 
the .10 level. The Internal Locus of Control scale effect size was .304 or “small.” 

Self-Esteem scale score results are presented last in Table 3. The possible points could 
range from 1 0 to 60. At pretest, the mean was 42.3 1 and the standard deviation was 9.86. The 
pretest coefficient of variation was .233 and the standard error of the mean was 1 .74 (the highest 
such value in the table). At posttest, the mean increased 1.85 points to 44.16, while the standard 
deviation decreased slightly to 9.50. Both the coefficient of variation and the standard error of 
the mean decreased slightly at posttest to .21 5 and 1 .68, respectively. The t-value was 2.22, 
which was significant at the .05 level. The Self-Esteem scale effect size was .1 88 or “less than 
small.” 



Parenting Materials Evaluation 



When comparing responses across the four instruments related to the five evaluative 
criteria (interest, understandable, informative, utility, and attractiveness), several comments can 
be made. First, most parents felt that the majority of other parents would find the materials 
interesting and would be willing to do the activities with their own children. Second, most of 
the parents felt the materials were not hard to understand, and they did not have suggestions for 
making them more helpful. Third, the materials were informative. Parents reported gaining 
knowledge of literacy education with their children. Fourth, most of parents indicated wanting to 
complete most or all of the suggested activities with their children, with just a few noting 
activities they would not do. Fifth, the evaluator judged the materials to be rather bland and 
unappealing in their field-test form. Although the content was quite satisfactory, based on 
parents’ responses, the attractiveness of the materials could be improved greatly. 
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CONCLUSIONS 



The two main purposes of this evaluation were (1) to evaluate the impact of the 
Monongalia County Even Start Program on a sample of families in terms of adult and parenting 
education outcomes, and (2) to evaluate the program’s parenting education literacy materials. 
Drawing on findings presented earlier, conclusions of the evaluation are presented in this 
section. These conclusions are presented in the same order as the findings. 



Treatment Months 

We conclude that the average Even Start treatment period for the ABLE instrument was 
almost two program years, while the average treatment period for the Parent Index instrument 
was about six months. Also, there was much more variation in the treatment period for the 
ABLE instrument than for the Parent Index instrument. 



Reading Comprehension 

From these findings, we conclude that the Monongalia County Even Start Program had 
some impact on parents ’ Reading Comprehension. 



Number Operations 

From these results, we conclude that the Monongalia County Even Start Program had a 
moderate impact on parents ’ Number Operations. 



Problem Solving 

From these positive findings, we conclude that the Monongalia County Even Start Program 
had a very large impact on parents ’ Problem Solving. 



Total Mathematics 

From these positive findings, we conclude that the Monongalia County Even Start Program 
had a large impact on parents ’ Total Mathematics. 
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Given the results of the four ABLE instrument scales, we conclude further that the 
Monongalia County Even Start Program had much more impact in the mathematics area than in 
reading comprehension. 



Parent Index Reliabilities 

Based on the Cronbach Alpha reliabilities for the four scales of the Parent Index 
instrument with a small sample of Monongalia County Even Start Program parents, we conclude 
that the Parent Index instrument possesses satisfactory internal consistency reliability. 



Parenting Efficacy 

From these findings, we conclude that the Monongalia County Even Start Program had no 
impact on the adults ’ Parenting Efficacy. 



Parental Responsibility 

From these results, we conclude that the Monongalia County Even Start Program had a 
little impact on the adults ’ Parental Responsibility. 



Internal Locus of Control 

From this information, we conclude that the Monongalia County Even Start Program had 
some impact on parents ’ Internal Locus of Control. 



Self-Esteem 

Based on these results, we conclude that the Monongalia County Even Start Program had 
some impact on parents ’ Self-Esteem. In fact, this scale had the most impact of the four PI 
scales. 

Given the results of the four PI instruments scales, we conclude further that the 
Monongalia County Even Start Program had the most impact on adults in the area of self 
esteem, followed by internal locus of control, then parental responsibility. 
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Parenting Materials Evaluation 

We conclude from the field test of the four parenting education literacy materials, 
developed by the Monongalia County Even Start Program staff, that they are very good in terms 
of: (1) interesting, (2) understandable, (3) informative, and (4) utility. However, we conclude 
that the materials are poor in term of their attractiveness, but that they could be improved 
rather easily. 
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RECOMMENDATIONS 



Recommendations based on the findings and conclusions of this evaluation are offered to 
the Monongalia County Even Start Program staff. These recommendations are not in priority 
order. 

Given the positive outcomes on the four ABLE adult education outcome scales (two 
“large” one “moderate,” and one “some”), the Monongalia County Even Start Program should 
continue to offer its adult education component in the future. This evaluation documented 
sufficient adult education outcomes with this sample of Even Start parents to warrant its 
continuation. 

Related to above, the Even Start staff may want to discuss the ABLE reading 
comprehensive outcomes being lower than the three mathematics scale outcomes. Thoughtful 
discussions by staff might result in reasonable explanations for the reading comprehension 
results or action steps to try to change that result. 

The use of the new Parent Index instrument with the sample of parents in this evaluation 
yielded satisfactory Cronbach Alpha internal consistency reliabilities. The recommendation is to 
continue to administer the Parent Index instrument in future years and also to continue to 
compute the Cronbach Alpha reliabilities periodically. Also, Even Start staff should search for 
other projects that have used this instrument in order to locate group scale scores to use for 
comparative purposes. 

Based on the positive outcomes on three of the parenting education outcome scales, the 
Monongalia County Even Start Program should continue to offer its parent education component 
in the future. Sufficient outcome evidence for the parenting education component was revealed 
in this evaluation to merit its continuation. 

In addition, the Even Start staff may want to study the outcomes of the PI Parenting 
Efficacy scale in more detail. This should include a discussion of the scale’s items in relation to 
what is delivered by the Family Educators in their home visits. It may be that plausible, rival 
hypotheses explain the decrease in the Parenting Efficacy scale. For example, one alternative 
hypothesis for the lower scores could be that, as a result of the parenting education component, 
parents broadened and/or heightened their expectations for dealing with their children, resulting 
in their lower posttest scores. 

With this second evaluation completed. Even Start staff have evidence of positive 
outcomes of two different instruments for each of the two areas of adult education and parenting 
education. Thus, there is no clear “best” instrument for either component. This is a decision for 
the staff to make. This recommendation calls for the Monongalia County Even Start Program 
staff to review, compare, and discuss the results of the two adult education component 
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instruments (CASAS and ABLE) and the two parenting education component instruments (HSQ 
and PI) in relation to other selection variables, e.g., cost, ease of administration, administration 
time required, relation to curriculum/services, ease of scoring, etc. The outcome of these 
deliberations by staff might be to select one instrument per component for use in the future or to 
use multiple instruments per component. 

It is recommended that Even Start staff continue to develop and test their original parenting 
education literacy materials. This first field test showed these four learning materials to be 
interesting, understandable, informative, and useful to the Even Start parents. However, they 
were judged by the evaluator to be unattractive and rather bland. This recommendation is for the 
staff to make the materials more appealing and attractive to the end users. This could be 
accomplished through the use of color stock, the addition of relevant illustrations or artwork, and 
a more professional typeface and layout. Also, these revised materials should contain the name 
of the agency. 

In addition, if the four parenting education literacy materials are revised, then it is 
recommended that another field test be conducted, with a larger sample of parents. Refinement 
based on that second field test should put the materials in final form for dissemination and use. 

Finally, the Monongalia County Even Start Program staff have a history of assessing its 
processes, outputs, and outcomes with independent evaluations. These independent evaluations 
are aided greatly by the commitment to and accuracy of data collection, scoring, reporting, and 
record keeping by program staff. This recommendation is for staff to maintain that tradition in 
future years so as to be in a position to solicit/conduct new evaluations of their Even Start 
Program and to continue improving their documentation and services. 
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